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the seventh left pixel in Fig. 1 9 corresponding to the third shutter time/v since the time 
of shutter opening and to the foreground component of the eighth left pixel in Fig. 1 9 
corresponding to the fouith shutter time/v since the time of shutter opening. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at an equal speed, with the movement quantity v being 4, the 
first foreground component F02/v of the sixth left pixel in Fig. 19, with the first shutter 
time/v since the time of stutter opening, is equal to the second foreground component 
of the seventh left pixel in Fig. 1 9 corresponding to the second shutter time/v since the 
time of shutter opening. Similarly, the foreground component F02/v is equal to the 
foreground component o:f the eighth left pixel in Fig. 19 corresponding to the third 
shutter time/v since the time of shutter opening. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at an equal speed, with the movement quantity v being 4, the 
first foreground component F03/v of the seventh left pixel in Fig. 19, with the first 
shutter time/v since the time of shutter opening, is equal to the second foreground 
component of the eighth le ft pixel in Fig. 1 9 corresponding to the second shutter time/v 
since the time of shutter opening. 

Although the number of times of the virtual splitting is four in the description 
with respect to Figs. 17 to 19, the number of times of the virtual splitting corresponds 
to the movement quantity v. The movement quantity v generally coiresponds to the 
movement speed of the object corresponding to the foreground. For example, if the 
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object corresponding to the foreground is moving so as to be displayed four pixels 
rightwards in a frame next to a previous reference frame, the movement quantity v is 
4. The number of times of the virtual splitting is set to 4 in association with the 
movement quantity v. Similarly, if the object corresponding to the foreground is 
moving so as to be displayed six pixels rightwards in a frame next to a previous 
reference frame, the movement quantity v is 6, with the number of times of the virtual 
splitting being six. 

Figs. 20 and 2 1 show the relation between the foreground area, background area, 
and the mixed area, comprised of the covered background area and the uncovered 
background area, on one hand, and the foreground and background components 
corresponding to the split shutter time, on the other hand, as described above. 

Fig.20 shows an example of extraction of pixels of the foreground area, 
background area and the mixed area as extracted from a picture corresponding to an 
object moving before a still background. In the embodiment shown in Fig.20, an 
object corresponding to the foreground is moving horizontally with respect to the 
picture. 

The frame #n+l is a frame next to the frame #n, with the frame #n+2 being a 
frame next to the frame 1 . 

Fig. 21 diagrammatically shows a model obtained on extracting pixels of the 
foreground area, background area and the mixed area, extracted in turn from one of 
the frames #n to #n+2, with the movement quantity v being 4, and on expanding the 
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pixel values of the extrac ted pixels along the time axis direction. 

Since the object corresponding to the foreground is moved, the pixel values of 
the foreground area are constituted by four different foreground components 
corresponding to the period of the shutter time/v. For example, the leftmost one of 
pixels of the foreground area shown in Fig,2 1 are FOl/v, F02/v, F03/v and F04/v. That 
is the pixels of the foreground area contain are corrupted with motion blurring. 

Since the object corresponding to the background is at a standstill, the light 
corresponding to the back;»round input to the sensor 1 1 during the time corresponding 
to the shutter time is not changed In this case, the pixel values of the background are 
free of the motion blurring. 

The pixel values or the pixels belonging to the mixed area composed of the 
covered background area or the uncovered background area are comprised of the 
foreground and background components. 

Amodel comprised of neighboring pixels in a row in plural frames, in which the 
pixel values of pixels lying at the same position on a frame are developed in the time 
axis direction, with the picture corresponding to an object being moved, is explained. 
For example, if the picture corresponding to the object is moving horizontally with 
respect to the picture, the pixels arrayed on the same row on the picture may be 
selected as the pixels in a row in a picture. 

Fig.22 diagrammatical ly shows a model obtained on temporally expanding the 
pixel values of pixels arrayed in a row of each of three frames of a picture of a 
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photographed object corresponding to a still background, with the developed pixels 
being at the same positions on the respective frames. The frame #n is the frame next 
to the frame #n-l, with .iie frame #n+l being the frame next to the frame #n. The 
remaining frames are termed in similar manner. 

The pixel values of B01 to B12 shown in Fig.22 are those of pixels 
corresponding to the object of the still background. Since the objet corresponding to 
the background is at a standstill, the pixel values of the corresponding pixels in the 
frames #n- 1 ti frame n+1 are not changed. For example, the pixel in the frame #n and 
the pixel in the frame #n+l , corresponding to the positions of the pixels having pixel 
values of BOS in the frame #n- 1, are of pixel values of BOS. 

Fig.23 shows pixel values of neighboring pixels in a row in each of three frames 
of a photographed picture of an object corresponding to the foreground moving 
rightwards in Fig.23, along with the object corresponding to the still background, with 
the pixel values being shown developed along the time axis direction. The model 
shown in Fig.23 includes a covered background area. 

In Fig.23, the object corresponding to the foreground is a rigid body and may 
be assumed to be moving at a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame. So, the movement quantity v of the foreground is 4, with the number of times 
of the virtual splitting beiig 4. 

For example, the foreground component of the leftmost pixel of the frame #n* 1 
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in Fig.23, with the first shutter time /v since the opening of the shutter, is F12/v, whilst 
the foreground component of the second left pixel, with the second shutter time /v 
since the opening of the shutter, is also F12v. The foreground component of the third 
left pixel in Fig.23, with the third shutter time/v since the opening of the shutter, and 
the foreground compone nt of the fourth left pixel in Fig.23, with the fourth shutter 
time/v since the opening of the shutter, are each F12/v. 

For example, the f Dreground component of the leftmost pixel of the frame #n- 1 
in Fig.23, with the second shutter time /v since the opening of the shutter, is Fl 1/v, 
whilst the foreground component of the second left pixel, with the third shutter time 
/v since the opening of the shutter, is also Fl 1 v. The foreground component of the 
third left pixel in Fig.23, with the fourth shutter time/v since the opening of the shutter, 
isFll/v. 

The foreground component of the leftmost pixel of the frame #n-l in Fig.23, 
with the third shutter time /v since the opening of the shutter, is FlO/v, whilst the 
foreground component of the second left pixel, with the fourth shutter time /v since 
the opening of the shutter, is also FlOv. The foreground component of the leftmost 
pixel in Fig.23, with the fo urth shutter time/v since the opening of the shutter, is F09/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the second left pixel of the frame #n-l in Fig.23, with the 
first shutter time/v as from the shutter opening time, is B0 1/v. The background 
component of the third left pixel of the frame #n- 1 in Fig.23, with the first and second 
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shutter time/v as from the shutter opening time, is B02/v, while the background 
component of the fourth left pixel of the frame #n«l in Fig.23, with the first to third 
shutter time/v as from the shutter opening time, is B03/v. 

In the frame #n-l in Fig.23, the leftmost pixel belongs to the foreground area, 
while the second to fouith left pixels belong to the mixed area which is the covered 
background area. 

The fifth to twelfth left pixels of the frame #n-l in Fig.23 belong to the 
background area, with the corresponding pixel values being B04 to B 1 1 , respectively. 

The first to fifth pixels of the frame fori in Fig.23 belong to the background 
area. The foreground component in the foreground area of the frame #n, with the 
shutter time/v, is one of F05v to F12/v. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame, the foreground component of the fifth left pixel of the frame #n-in Fig.23, with 
the first shutter time /v since the opening of the shutter, is F12/v, whilst the foreground 
component of the sixth lert pixel, with the second shutter time /v since the opening of 
the shutter, is also F12v. The foreground component of the seventh left pixel in 
Fig.23, with the third shutter time/v since the opening of the shutter, and the 
foreground component of the eighth left pixel in Fig.23, with the fourth shutter time/v 
since the opening of the shutter, are each F12/v. 
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The foreground component of the fifth left pixel of the frame #n in Fig.23, with 
the second shutter time /v since the opening of the shutter, is Fll/v, whilst the 
foreground component of the sixth left pixel, with the third shutter time /v since the 
opening of the shutter, is also Fllv. The foreground component of the seventh left 
pixel in Fig.23, with the fourth shutter time/v since the opening of the shutter, is Fl 1/v. 

The foreground component of the fifth left pixel of the frame #n in Fig.23, with 
the third shutter time /v since the opening of the shutter, is FlO/v, whilst the 
foreground component of the sixth left pixel, with the fourth shutter time /v since the 
opening of the shutter, is also FlOv. The foreground component of the fifth left pixel 
in Fig.23, with the fourth shutter time/v since the opening of the shutter, is F09/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the sixth left pixel of the frame #n in Fig.23, with the first 
shuttertime/vasfrommeshutteropemrjgtime,i S B05/v. The background component 
of the seventh left pixel of the frame #n in Fig.23, with the first and second shutter 
time/v as from the shutter opening time, is B06/v, while the background component 
of the eighth left pixel of ihe frame #n in Fig.23, with the first to third shutter time/v 
as from the shutter opening time, is B07/v. 

In the frame #n-l in Fig.23, the first to ninth left pixels belong to the foreground 
area, while the sixth to eighth left pixels belong to the mixed area which is the covered 
background area. 

The first to ninth to twelfth left pixels of the frame #n+l in Fig.23 belong to the 
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foreground area, with the pixel values being B08 to Bl 1, respectively. 

The first to ninth pixels of the frame #n+l in Fig.23 belong to the foreground 
area. The foreground component in the foreground area of the frame #n+l, with the 
shutter time/v, is one of F01 v to F12/v. 

Since the object c orresponding to the foreground is a rigid body and may be 
assumed to be moving at a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame, the foreground component of the ninth left pixel of the frame #n+l in Fig.23, 
with the first shutter time fy since the opening of the shutter, is F12/v, whilst the 
foreground component of the tenth left pixel, with the second shutter time /v since the 
opening of the shutter, is also F12v. The foreground component of the eleventh left 
pixel in Fig.23, with the tliird shutter time/v since the opening of the shutter, and the 
foreground component of the twelfth left pixel in Fig.23, with the fourth shutter time/v 
since the opening of the shutter, are each F12/v. 

The foreground component of the ninth left pixel of the frame #n+l in Fig.23, 
with the second shutter time /v since the opening of the shutter, is Fll/v, whilst the 
foreground component of the tenth left pixel, with the third shutter time N since the 
opening of the shutter, is also Fl 1 v. The foreground component of the eleventh left 
pixel in Fig.23, with the fourth shutter time/v since the opening of the shutter, is Fl 1/v. 

The foreground component of the ninth left pixel of the frame #n+l in Fig.23, 
with the third shutter time /v since the opening of the shutter, is FlO/v, whilst the 
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foreground component of the tenth left pixel, with the fourth shutter time /v since the 
opening of the shutter, is also F 1 Ov. The foreground component of the ninth left pixel 
of the frame #n+lin Fig.23, with the fourth shutter time/v since the opening of the 
shutter, is F09/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the tenth left pixel of the frame #n+l in Fig.23, with the 
first shutter time/v as fiom the shutter opening time, is B09/v. The background 
component of the eleventh left pixel of the frame #n+l in Fig.23, with the first and 
second shutter time/v as from the shutter opening time, is BlO/v, while the background 
component of the twelfth left pixel of the frame #n+l in Fig.23, with the first to third 
shutter time/v as from the shutter opening time, is B 1 1/v. 

In the frame #n+I in Fig.23, the tenth to twelfth left pixels correspond to the 
mixed area which is the covered background area. 

Fig.24 diagrammat ically shows apicture obtained on extracting the foreground 
component from the pixel values shown in Fig.23. 

Fig.25 shows neighboring pixels in a row of each of three frames of a 
photographed picture ofthe foreground corresponding to an object moving rightwards 
in the drawing, along with the still background. Li Fig.25, there is also shown the 
uncovered background area. 

In Fig.25, the objeci: corresponding to the foreground is a rigid body and may 
be assumed to be moving at a constant speed, with the foreground picture beingmoved 
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so that the foreground picture will be displayed four pixels rightwards in the next 
frame. So, the movement quantity v of the foreground is 4. 

For example, the foreground component of the leftmost pixel of the frame #n- 1 
in Fig.25, with the first shutter time /v since the opening of the shutter, is F13/v, whilst 
the foreground component of the second left pixel, with the second shutter time /v 
since the opening of the shutter, is also F13v. The foreground component of the third 
left pixel in Fig.23, with the second shutter time/v since the opening of the shutter, and 
the foreground component of the fourth left pixel in Fig.25, with the fourth shutter 
time/v since the opening of the shutter, are each F13/v. 

For example, the foreground component of the second left pixel of the frame 
#n* 1 in Fig.23, with the first shutter time /v since the opening of the shutter, is F14/v, 
whilst the foreground component of the third left pixel, with the second shutter time 
/v since the opening of die shutter, is also F14v. The foreground component of the 
third left pixel in Fig.25, with the first shutter time/v since the opening of the shutter, 
isFlS/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the leftmost pixel of the frame #n-l in Fig.25, with the 
second to fourth shutter time/v as from the shutter opening time, is BOl/v. The 
background component of the second left pixel of the frame #n*l in Fig.25, with the 
third and fourth shutter time/v as from the shutter opening time, is B26/v, while the 
background component of the third left pixel of the frame fori in Fig.25, with the 



PAGE 11/33 * RCVD AT 7/16/2009 1:02:59 PM [Eastern Daylight Time] " SVR:USPTO-EFXRF-6/19 " DNIS:2738300 * CSID:12125880500* DURATION (mm-ss): 23-24 



07/16/2009 13:09 FAX 12125880500 



FROMMER LAWRENCE & HAUG 



©012/051 



61 

fourth shutter time/v as from the shutter opening time, is B27/v. 

In the frame #n- 1 in Fig.25, the first to third left pixel belongs to the mixed area 
which is the covered bac kground area. 

The fourth to twdfth left pixels of the frame #n- 1 in Fig.25 belong to the 
foreground area, with the foreground component of the foreground of the frame being 
one of F13vto F24v. 

The first to fourth left pixels of the frame #n in Fig.25 belong to the background 
area, with the pixel value s being B25 to B28, respectively. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at a constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame, the foreground component of the fifth left pixel of the frame #n in Fig.23, with 
the first shutter time /v sin ce the opening of the shutter, is F 13/v, whilst the foreground 
component of the sixth left pixel, with the second shutter time /v since the opening of 
the shutter, is also F13v. The foreground component of the seventh left pixel in 
Fig.25, with the third shutter time/v since the opening of the shutter, and the 
foreground component of the eighth left pixel in Fig.25, with the fourth shutter time/v 
since the opening of the shutter, are each F13/v. 

The foreground component of the sixth left pixel of the frame #n in Fig.23, with 
the first shutter time /v since the opening of the shutter, is F 1 4/v, whilst the foreground 
component of the seventh left pixel, with the second shutter time /v since the opening 
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of the shutter, is also F14v. The foreground component of the eighth left pixel in 
Fig.25, with the first shu rter time/v since the opening of the shutter, is F15/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the fifth left pixel of the frame #n in Fig.25, with the second 
to fourth shutter time/v as from the shutter opening time, is B29/v. The background 
component of the sixth left pixel of the frame #n in Fig.25, with the third and fourth 
shutter time/v as from the shutter opening time, is B30/v, while the background 
component of the seventh left pixel of the frame #n in Fig.23, with the fourth shutter 
time/v as from the shutte r opening time, is B31/v. 

In the frame #n in Fig.25, the first to ninth left pixels belong to the foreground 
area, while, the fifth to seventh left pixels belong to the mixed area which is the 
covered background area 

The eighth to tweith left pixels of the frame #n-f-l in Fig.25 belong to the 
foreground area, with the pixel values being B25 to B32, respectively. 

The first to eighth pixels of the frame #n+l in Fig.25 belong to the background 
area, with the pixel value!! being B25 to B32, respectively. 

Since the object corresponding to the foreground is a rigid body and may be 
assumed to be moving at ;i constant speed, with the foreground picture being moved 
so that the foreground picture will be displayed four pixels rightwards in the next 
frame, the foreground coriponent of the ninth left pixel of the frame #n+l in Fig.25, 
with the first shutter time; /v since the opening of the shutter, is F13/v, whilst the 



PAGE 13/33 ' RCVD AT 7/16/2009 1:02:59 PM [Eastern Daylight Time] ' SVR:USPTO-EFXRF-6/19 • DNIS: 2738300 * C8ID: 121 25880500 * DURATION (mm-ss): 23-24 



07/16/2009 13:10 FAX 12125880500 FROMMER LAWRENCE & HAUG ©014/051 



63 

foreground component of the tenth left pixel, with the second shutter time /v since the 
opening of the shutter, is also F13v. The foreground component of the eleventh left 
pixel in Fig.25, with the third shutter time/v since the opening of the shutter, and the 
foreground component of the twelfth left pixel in Fig.25, with the fourth shutter time/v 
since the opening of the shutter, are each F13/v. 

The foreground component of the tenth left pixel of the frame #n+l in Fig.25, 
with the first shutter time /v since the opening of the shutter, is F14/v, whilst the 
foreground component of the eleventh left pixel, with the second shutter time /v since 
the opening of the shutter, is also F14v. The foreground component of the twelfth left 
pixel in Fig.25, with the first shutter time/v since the opening of the shutter, is F15/v. 

Since the object corresponding to the background is at a standstill, the 
background component of the ninth left pixel of the frame #n+l in Fig.25, with the 
second to fourth shutter time/v as from the shutter opening time, is B33/v. The 
background component of the tenth left pixel of the frame #n+l in Fig.25, with the 
third and fourth shutter time/v as from the shutter opening time, is B34/v, while the 
background component of the eleventh left pixel of the frame #n+l in Fig.25, with the 
fourth shutter time/v as from the shutter opening time, is B35/v. 

In the frame #n+l :ji Fig.25, the ninth to eleventh left pixels correspond to the 
mixed area which is the covered background area. 

In Fig.25, the twelfth left pixel of the frame #n+l belong to the foreground area. 
The foreground component with the shutter time/v in the foreground area of frame 
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#n+I is one of F13v to F16v. 

Fig.26 diagrammaticalry shows a picture obtained on extracting the foreground 
component from the pixel values shown in Fig.25. 

Reverting to Fig. 10, the area specifying unit 103 associates a flag, indicating 
that a given picture belong to the foreground area, a background area, a covered 
background area or an uncovered background area, from pixel to pixel, using pixel 
value of plural frames, aid routes the resulting areal information to the mixing ratio 
calculating unit 104 and to the motion blurring adjustment unit 106. 

Based on the pixel values of plural frames and the areal information, the mixing 
ratio calculating unit 104 computes the mixing ratio a for each of the pixels contained 
in the mixed area, and sends the computed mixing ratio a to the 
foreground/background separating unit 105. 

Based on the pixel values of the plural frames, areal information and the mixing 
ratio a, the foreground/background separating unit 105 extracts the foreground 
component picture made up only of the foreground component to send the extracted 
component picture to the motion blurring adjustment unit 106. 

Based on the foreground component picture sent from the 
foreground/background separating unit 105, the motion vector sent from the motion 
detection unit 102 and on the areal information sent from the area specifying unit 103, 
the motion blurring adjuslment unit 106 adjusts the quantity of the motion blurring 
contained in the foreground component picture to output the foreground component 
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picture adjusted for the motion blurring. 

Referring to the flowchart of Fig. 27, the processing for adjusting the motion 
blurring caused by the signal processor 12 is explained. At step S101, the area 
specifying unit 103 executes the area specifying processing for generating the areal 
information indicating to which of the foreground area, background area, covered 
background area or the uncovered background area belong the pixels of the input 
picture, from one pixel of the input picture to another. The area specifying processing 
will be explained subsequently by referring to the flowchart of Fig.36. The area 
specifying unit 103 sends the generated area information to the mixing ratio 
calculating unit 104. 

Meanwhile, the area specifying unit 103 at step S101 may generate the areal 
information indicating to which of the foreground area, background area or the mixed 
area belong the pixels of the input picture, from one pixel of the input picture to 
another, based on the input picture. In this case, no distinction is made between the 
covered background area and the uncovered background area. In this case, the 
foreground/background separating unit 105 and the morion blurring adjustment unit 
106 decide whether the mixed area is the covered background area or the uncovered 
background area, based on the direction of the motion vector. For example, if the 
foreground area, mixed erea and the background area are arrayed sequentially in 
association with the direction of the motion vector, the mixed area is verified to be the 
covered background area, whereas, if the background area, mixed area and the 
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foregroundarea are airayed sequentially in association with the direction of the motion 
vector, the mixed area is verified to be the uncovered background area. 

At step S102, the mixing ratio calculating unit 104 calculates the mixing ratio 
a, from one pixel contained in the mixing area to another, based on the input picture 
and the area informatioa. The processing for computing the mixing ratio will be 
explained in detail subsequently by referring to the flowchart of Fig.46. The mixing 
ratio calculating unit 104 sends the computed mixing ratio a to the 
foreground/background Jteparating unit 105. 

At step S103, the foreground/background separating unit 105 extracts the 
foreground component from the input picture, based onthemotion vector and theareal 
information, to send the extracted component to the motion blurring adjustment unit 
106 as the foreground component picture. 

At step S104, the motion blurring adjustment unit 106 generates a processing 
unit for indicating a posidon on the picture of pixels arrayed consecutively in the 
movement direction of each of the uncovered background area, foreground area and 
the covered background area, based on the motion vector and on the area information, 
to adjust the quantity of th e motion blurring contained in the foreground component 
corresponding to the processing unit The processing for adjusting the quality of the 
motion blurring will be explained subsequently byreferring to the flowchart of Fig.63. 

At step S 105, the signal processor 12 verifies whether ornot the processing has 
been finished for the entirs picture. If the signal processor 12 has verified that the 
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processing has not been finished fort the entire picture, it proceeds to step SI 04 to 
repeat the processing for adjusting the quantity of the motion blurring for the 
foreground component corresponding to the processing unit. 

If, at step S106, it is verified that the processing has been finished for the entire 
picture, the processing is terminated. 

In this manner, the signal processor 12 is able to separate the foreground and 
the background from each other to adjust the quantity of the motion blurring contained 
in the foreground. Thai: is, the signal processor 12 is able to adjust the amount of 
motion blurring contained in sample data as pixel value of the foreground pixel. 

In the following, illustrative structures of the area specifying unit 103, mixing 
ratio calculating unit 1 04 : foreground/background separating unit 105 and the motion 
blurring adjustment unit 106 are hereinafter explained. 

Fig.28 is a block diagram showing an illustrative structure of the area specifying 
unit 103. A frame memory 121 stores an input picture on the frame basis. When a 
frame being processed is a frame #n, the frame memory 121 stores a frame #n-2, as 
a frame two frames before the frame #n, a frame #irl,asa frame one frame before 
the frame #n, a frame #n+l, as a frame one frame after the frame #n, and a frame 
#n+2, as a frame two frames after the frame #n. 

A stall/movement thscrirmnating unit 122-1 reads out a pixel value of a pixel 
of the frame #n+2 lying at the same position as the position on the picture of the pixel 
of the frame #n being area-specified, and a pixel value of a pixel of the frame #n+l 
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lying at the same position as the position on the picture of the pixel of the frame #n 
being area-specified, frcm the frame memory 12 1, to calculate an absolute value of the 
difference of the read-our pixel values. The still/movement discriminating unit 122-1 
verifies whether or not the absolute value of the difference between the pixel value of 
the frame #n+2 and the frame #n+ 1 is larger than a predetermined threshold value Th. 
If it is verified that the absolute value of the difference is larger than the threshold 
value Th, the still/movement discriminating unit 122-1 routes a still/movement 
decision specifying the movement decision to an area decision unit 123-1. If it is 
verified that the absolute value o the difference between the pixel value of the frame 
#a+2 nd the pixel value of the frame #n+l is not larger than the threshold value Th, 
the still/movement discriminating unit 122-1 routes a still/movement decision 
specifying the still decision to an area decision unit 123-1. 

A still/movement discriminating unit 122-2 reads out a pixel value ofa pixel of 
the frame #n+l lying at th 2 same position as the position on the picture of the pixel of 
the frame #n being area-specified, and a pixel value ofa pixel of the frame #n+l lying 
at the same position as the position on the picture of the pixel of the frame #n being 
area-specified, from the frame memory 121, to calculate an absolute value of the 
difference of the read-our pixel values. The still/movement discriminating unit 122-2 
verifies whether or not the absolute value of the difference between the pixel value of 
the frame #n+l and the frame #n is larger than a predetermined threshold value Th. 
If it is verified that the absolute value of the difference between the is larger than the 
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threshold value Th, the still/movement discriminating unit 122-1 routes a 
still/movement decision specifying the movement decision to an area decision unit 
123-land to an area decision unit 123-2. If it is verified that the absolute value of the 
difference between the pixel value of the pixel of the frame #n + l and that of the pixel 
of the frame #n is not larger than the threshold value Th, the still/movement 
discriminatmgunit 122-1 routes a still/movement decision specifying the still decision 
to an area decision unit 123-1 and to an area decision unit 123-2. 

A still/movement discriminating unit 122-3 reads out a pixel value of a pixel of 
the frame #n lying at the same position as the position on the picture of the pixel of the 
frame #n being area-specified, and a pixel value of a pixel of the frame #n - 1 lying at 
the same position as the p osition on the picture of the pixel of the frame #n being area- 
specified, from the frame memory 1 2 1 , to calculate an absolute value of the difference 
of the read-our pixel values. The still/movement discrirninating unit 122-3 verifies 
whether or not the absolute value of the difference between the pixel value of the 
frame #n and the frame ft vl is larger than a predetermined threshold value Th. If it 
is verified that the absolute value of the difference between the pixel values is larger 
than the threshold value Th, the still/movement discrirninating unit 122-3 routes a 
still/movement decision specifying the movement decision to an area decision unit 
123-landtoanareadecisionunit 123-3. If it is verified that the absolute value of the 
difference between the pix*l value of the pixel of the frame #n and that of the pixel of 
the frame #n l is not larger than the threshold value Th, the still/movement 
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discriminating unit 122- 3 routes a still/movement decision specifying the still decision 
to an area decision unit 123-2 and to an area decision unit 123-3. 

A still/movement: discriminating unit 122-4 reads out the pixel value of the pixel 
of the frame #n-l lying at the same position as the position on the picture of the pixel 
of the frame #n heing aiea-specified, and the pixel value of the pixel of the frame #n-2 
lying at the same position on the picture of the pixel of the frame #n being area- 
specified, to calculate the absolute value of the difference of the pixel values. The 
still/movement discriminating unit 122-4 verifies whether or not the absolute value of 
the difference of the pixel value of the frame #n-l and the pixel value of the frame 
#n-2 is larger than the predetermined threshold value Th. If the absolute value of the 
difference between the pixel value of the frame #n- 1 and the pixel value of the frame 
#n-2 is verified to be larger than the threshold value Th, a still/movement decision 
indicating the decision for movement is routed to the area decision unit 123-3. If it is 
verified that the absolute value of the difference between the pixel value of the frame 
#n- 1 and the pixel value of the frame #n-2 is not larger than the threshold value Th, the 
still/movement discriminating unit 122-4 routes a still/movement decision indicating 
the still decision to the area decision unit 123-3. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-1 indicates still and the still/movement decision routed from the 
still/movement discrimimting unit 122-2 indicates movement, the area decision unit 
123-1 decides that the pixel on the frame #n being area-specified belongs to the 
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uncovered background area and sets "1" in an uncovered background area decision 
flag associated with the pixel being area-specified for indicating that the pixel belongs 
to the uncovered backgi ound area. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-1 indicates movement and the still/movement decision routed from the 
still/movement discriminating unit 122-2 indicates still, the area decision unit 123-1 
decides that the pixel oa the frame #n being area-specified does not belong to the 
uncovered background erea and sets "0" in an uncovered background area decision 
flag associated with the pixel being area-specified for indicating that the pixel does not 
belong to the uncovered background area. 

The area decision unit 123-1 routes the uncovered background area decision 
flag, having "1" or "0" set in this manner, to a decision flag storage memory 124. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-2 indicates still and the still/movement decision routed from the 
still/movement discriminating unit 122-3 indicates still, the area decision unit 123-2 
decides that the pixel on the frame #n being area-specified belongs to the still area and 
sets "1" in a still area decision flag associated with the pixel being area-specified for 
indicating that the pixel belongs to the uncovered background area. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-2 indicates movement or the still/movement decision routed from the 
still/movement discriminadng unit 122-3 indicates movement, the area decision unit 
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123-2 decides that the pixel on the frame #n being area-specified does not belong to 
the still area and sets "0" in a still area decision flag associated with the pixel being 
area-specified for indicating that the pixel does not belong to the still area. 

The area decision unit 123-2 routes the still area decision flag, thus having " 1 " 
or "0" set therein, to the decision flag storage memory 124. 

If the still/movement decision routed from the still/movement discriininating 
unit 122-2 indicates movement and the still/movement decision routed from the 
still/movement discriminating unit 122-3 indicates movement, the area decision unit 
123-2 decides that the pixel on the frame #n being area-specified belongs to the 
movement area and sets " 1 " in a movement area decision flag associated with the pixel 
being area-specified for ; ndicating that the pixel belongs to the movement area 

If the still/movement decision routed from the still/movement discriminating 
unit 1 22-2 indicates still or the still/movement decision routed from the still/movement 
discriminating unit 122-3 indicates still, the area decision unit 123-2 decides that the 
pixel on the frame #n being area-specified does not belong to the movement area and 
sets "0" in a movement area decision flag associated with the pixel being area- 
specified for indicating that the pixel does not belong to the movement area. 

The area decision unit 123-2 routes the movement area decision flag, thus 
having "1" or "0" set therein, to the decision flag storage memory 124. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-3 indicates movement and the still/movement decision routed from the 
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still/movement discriminating unit 122-4 indicates still, the area decision unit 123-3 
decides that the pixel on the frame #n being area-specified belongs to the uncovered 
background area and sets "1" in a covered background area decision flag associated 
with the pixel being area -specified for indicating that the pixel belongs to the covered 
background area. 

If the still/movement decision routed from the still/movement discriminating 
unit 122-3 indicates still or the still/movement decision routed from the still/movement 
discriminating unit 122-4 indicates movement, the area decision unit 123-3 decides 
that the pixel on the frame #n being area-specified does not belong to the covered 
background area and sets "0" in a covered background area decision flag associated 
with the pixel being area specified for indicating that the pixel does not belong to the 
covered background are£>. 

The area decision unit 123-3 routes the covered background area decision flag, 
thus having " 1 " or "0" set therein, to the covered background area decision flag storage 
memory 124. 

The decision flag storage memory 124 stores the uncovered background area 
decision flag, sent from the area decision unit 123-1, the still area decision flag, sent 
from the area decision unit 123-2, the movement area decision flag, sent from the area 
decision unit 123-2, and the uncovered background area decision flag, sent from the 
area decision unit 123-3. 

The decision flag storage memory 124 sends the uncovered background area 
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decision flag, still area decision flag, movement area decision flag and the covered 
background area decision flag to a synthesis unit 125. Based on the uncovered 
background area decision flag, still area decision flag, movement area decision flag 
and the covered background area decision flag, supplied from the decision flag storage 
memory 124, the synthesis unit generates the area information indicating to which of 
the uncovered background area, still area, movement area and the covered background 
area belong the respectiv e pixels, and routes the information so generated to a decision 
flag storage frame memory 126. 

The decision flag storage frame memory 126 stores the area information, 
supplied from the synthssis unit 125, while outputting the area information stored 
therein. 

Referring to Figs.29 to 33, a typical processing by the area specifying unit 1 03 
is explained. 

When an object corresponding to the foreground is moving, the position of the 
picture corresponding to the object on the picture screen is changed from frame to 
frame. Referring to Fig.29, a picture corresponding to an object at a position Yn(x, y) 
in a frame #n is positioned at Yn+l(x,y) at the next frame #n+l. 

Fig.30 diagrammatically shows a model of pixel values of a row of pixels 
neighboring to one another along the moving direction of the picture corresponding 
to the foreground. For example, if the movement direction of the picture 
corresponding to the foreground is horizontal relative to the picture screen, the 
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diagrammatic view of Fig.30 shows a model in which pixel values of pixels 
neighboring to one another on one line are developed in the time axis direction. 
In Fig.30, the line in the frame #n is the same as one in the frame #n+l. 
The components of the foreground corresponding to the object contained in the 
second to the thirteenth pixels as counted from left in the frame #n are included in the 
sixth to seventeenth pixsls as counted from the left of the frame #n+l. 

The pixels belonging to the covered background area in the frame #n are the 
eleventh to thirteenth pixels as counted from left, whilst the pixels belonging to the 
uncovered background area are the second to fourth pixels as counted from left. The 
pixels belonging to the covered background area in the frame #n+l are the fifteenth 
to seventeenthpixelsascountedfromleft, whilst the pixels belonging to the uncovered 
background area are the sixth to eighth pixels as counted from left. 

In the example shown in Fig.30, since the foreground component in the frame 
#n are moved by four pixels in the frame #n+l, the movement quantity v is 4. The 
number of times of the virtual sphtting corresponds to the movement quantity and is 
equal to 4. 

The change in Ihe pixel values of pixels belonging to the mixed area ahead and 
at back of the frame being considered is explained. 

In the frame #n shown in Fig.31, in which the background is still and the 
movement quantity of the foreground v is 4, pixels belonging to the covered 
background area are fifteenth to seventeenth pixels from left. Since the movement 
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quantity v is 4, the fifteenth to seventeenth pixels from left in the directly previous 
frame tori contain only the background components and belong to the background. 
The fifteenth to seventeenth pixels from left in the further previous frame tor2 contain 
only the background components and belong to the background area. 

Since the object corresponding to the background is still, the pixel value of the 
fifteenth pixel from the left of the frame tori is not changed from the pixel value of 
the fifteenth pixel from the left of the frame tor2. Similarly, the pixel value of the 
sixteenth pixel from the left of the frame tor 1 is not changed from the pixel value of 
the sixteenth pixel from the left of the frame tor2, whilst the pixel value of the 
seventeenth pixel from tbeleft of the frame tori is not changed from the pixel value 
of the seventeenth pixel from the left of the frame tor2. 

That is, the pixels of the frame tori and the frame tor 2 corresponding to the 
pixels belonging to the covered background area in the frame toi are comprised only 
of the background components and are not changed, so that the absolute value of the 
difference is substantially 0. So, the still/movement decision on the pixels of the frame 
tor 1 and frame tor2 corresponding to the mixed area in the frame toi is made as being 
still by the still/moving discriminating unit 122-4. 

Since the pixels belonging to the covered background area in the frame #n 
contain the foreground components, the corresponding pixel values differ from those 
in which the pixels are comprised only of background components in the frame tori. 
Therefore, the pixels belonging to the mixed area in the frame toi and the 
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corresponding pixels of the frame #n-l are verified to be moving pixels by the 

still/moving discriminating unit 122-3. 

When fed with the result of still/movement decision indicating the movement 
from the still/moving discriminating unit 122-3 and with the result of still/movement 
decision indicating the still from the still/moving discriminating unit 122-4, the area 
decision unit 123-3 decides that the pixel in question belongs to the covered 
background area. 

The pixels contained in the uncovered background area in the frame #n in which 
the background is still and the movement quantity v of the foreground is 4 are second 
to fourth pixels as counied from left. Since the movement quantity v is 4, the second 
to fourth pixels from left in the next frame #n+l contain only the background 
components and belong to the background area. In the second next frame #n+2, the 
second to fourth pixels from left contain only the background components and belong 
to the background area. 

Since the object corresponding to the background is still, the pixel value of the 
second pixel from left of the frame #n+2 is not changed from the pixel value of the 
secondpixel from left of lie frame #n+l. Similarly, the pixel value of the second pixel 
from left of the frame #d+2 is not changed from the pixel value of the second pixel 
from left of the frame #n+l, whilst the pixel value of the third pixel from left of the 
frame #n+2 is not changed from the pixel value of the fourth pixel from left of the 
frame #n+l. 
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That is, the pixels of the frame #n+l and frame #n+2 corresponding to the 
pixels belonging to the uncovered background area in the frame #n are composed only 
of background compontaits and are not changed in the pixel values. So, the absolute 
value of the difference is: approximately zero. Therefore, the pixels of the frame #n+l 
and frame #n+2 corresponding to the pixels belonging to the mixed area in the frame 
#n are decided by the st 11/moving discriminating unit 122-1 to be still pixels. 

The pixels belonging to the uncovered background area in the frame #n contain 
the foreground components and hence differ in pixel values from the pixels in the 
frame #n+l composed only of the background components. So, the pixels belonging 
to the mixed area in the frame #n and those of the corresponding frame ftrl are 
decided by the still/moving discriminating unit 122-2 to be moving pixels. 

The area decision unit 123-1 is fed in this manner with the result indicating 
movement from the still/moving discriminating unit 122-2. If fed with the result 
indicating still from the stUl/moving discriminating unit 122-1, the area decision unit 
1 23- 1 decides that the corresponding pixel belongs to the uncovered background area. 

Fig.33 shows decision conditions of the area specifying unit 103 in the frame 
#n. When the pixel of the frame #n-2 at the same position as the position on the 
picture of the pixel of the frame #n being verified and the pixel of the frame #n- 1 at 
the same position as the position on the picture of the pixel of the frame #n being 
verified, are decided to be still, whilst the pixel of the frame #n- 1 at the same position 
as the position on the picture of the pixel of the frame #n being verified and the pixel 
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of the frame #n are decided to be moving, the area specifying unit 1 03 decides that the 
pixel of the frame #n being verified belongs to the covered background area. 

When the pixel of the frame tori at the same position as the position on the 
picture of the pixel of ihe frame #n being verified and the pixel of the frame #n are 
decided to be still, whilst the pixel of the frame #n and the pixel of the frame #n+l at 
the same position as the position on the picture of the pixel of the frame #n being 
verified are decided to be still, the area specifying unit 1 03 decides that the pixel of the 
frame #n being verified belongs to the still area. 

When the pixel of the frame fcrl at the same position as the position on the 
picture of the pixel of the frame #n being verified and the pixel of the frame #n are 
decided to be moving, wlrilst the pixel of the frame #n and the pixel of the frame #n+l 
at the same position as the position on the picture of the pixel of the frame #n being 
verified are decided to be still, the area specifying unit 1 03 decides that the pixel of the 
frame #n being verified belongs to the moving area. 

When the pixel of the frame #n and the pixel of the frame #n+l at the same 
position as the position o:a the picture of the pixel of the frame #n being verified are 
decided to be moving and when the pixel of the frame #n+ 1 at the same position as the 
position on the picture of the pixel of the frame #n being verified and the pixel of the 
frame #n+ 1 at the same position as the position on the picture of the pixel of the frame 
#n being verified and the pixel of the frame #n+2 at the same position as the position 
on the picture of the pixe l of the frame #n being verified are decided to be still, the 
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area specifying unit 103 decides that the pixel of the frame #n being verified belongs 
to the uncovered background area. 

Fig.34 shows an example of the area decision by the area specifying unit 103. 
In Fig.34A, a pixel decided to belong to the covered background area is shown in 
white. In Fig.34B, a p ixel decided to belong to the uncovered background area is 
shown in white. 

In Fig.34C, a pixel decided to belong to the moving area is shown in white. In 
Fig.34D, a pixel decided to belong to the still area is shown in white. 

Fig. 35 shows the :jrea information representing the mixed area, among the area 
information output by the decision flag storage frame memory 126, as picture. In 
Fig.35, the pixel decided to belong to the covered background area or the uncovered 
background area, that is to the mixed area, is shown in white. The area information 
indicating the mixed area, output by the decision flag storage frame memory 126, 
indicates a textured portion surrounded by an untextured portion in the foreground 
area and the mixed area. 

Referring to the flowchart of Fig.36, the processing for area identification by 
the area specifying unit 103 is explained. At step S121, the frame memory 121 
acquires pictures of the frame #n-2 to frame #n+2, inclusive the frame #n. 

At step S122, the still/moving discriminating unit 122-3 checks whether or not 
the pixels at the same position of the frame #n-l and the frame #n are still. If the 
pixels are decided to be slill, the program moves to step S123 where the still/moving 
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discriminating unit 122-2 checks whether or not the pixels at the same position of the 
frame #n and the frame #n+l are still. 

If, at step S123, the pixels at the same position of the frame #n and the pixel of 
the frame #n+l are decided to be still, the program moves to step S124 where the area 
decision unit 1 23-2 sets "1" in the still area decision flag corresponding to the pixel of 
the area being verified for indicating that the pixel belongs to the still area. The area 
decision unit 123-2 serds the still area decision flag to the decision flag storage 
memory 124. The program then moves to step S125. 

If at step SI 22 the pixels at the same position of the frame #nl and the frame 
#n are decided to be moving or if at step S123 the pixels at the same position of the 
frame #n and the frame #n+l are decided to be moving, the pixel of the frame #n does 
not belong to the still area, so the processing at stepS124 is skipped and the program 
moves to step SI 25. 

At step S125, the still/moving discriminatmg unit 122-3 checks whether or not 
the pixels at the same position of the frame #n-l and the frame #n are moving. If the 
pixels are decided to hs moving, the program moves to step S126 where the 
still/moving discriminatir g unit 122-2 decides whether or not the pixels at the same 
position of the frame #n end the frame #n+l are moving. 

If, at step S 126, the pixels at the same position of the frame #n and the pixel of 
the frame #n+l are decided to be moving, the program moves to step S127 where the 
area decision unit 123-2 sets "l"in the movung area decision flag corresponding to the 
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